Generating Images From Spoken Descriptions

Text-based technologies, such as text translation from one language to another, and image captioning, are gaining popularity. However, approximately half of the world's languages estimated be lacking a commonly used written form. Consequently, these cannot benefit text-based technologies. This paper presents 1) new speech technology task, i.e., speech-to-image generation (S2IG) framework which translates descriptions photo-realistic images 2) without using any information, thus allowing unwritten potentially this technology. The proposed framework, referred S2IGAN, consists embedding network relation-supervised densely-stacked generative model. learns embeddings with supervision corresponding visual information images. model synthesizes images, conditioned on produced by network, that semantically consistent spoken descriptions. Extensive experiments conducted four public benchmark databases: two databases in text-to-image tasks, CUB-200 Oxford-102 for we created synthesized descriptions, natural often field cross-modal learning Flickr8k Places. Results demonstrate effectiveness S2IGAN synthesizing high-quality semantically-consistent signal, yielding good performance solid baseline S2IG task.